Determining identity of a person in a digital image

ABSTRACT

Aspects of the current disclosure include systems and methods for identifying a person in a query image by comparing the query image with digital images in a face database. In embodiments, a query feature vector may be extracted from the query image and a set of candidate feature vectors may be extracted from a set of images in the face database. In embodiments, the distances between the query feature vector and the candidate feature vectors are calculated. A feature vector, which includes a set of shortest distances among the calculated distances and a distribution of the set of shortest distances, may be generated. In embodiments, the feature vector is input to a trained decision tree to determine whether the person in the query image is the same person associated with one of the set of shortest distances.

TECHNICAL FIELD

The present disclosure relates to identifying a person in a face image,more particularly, to systems and methods for determining identity of aperson in a query image by comparing the query image, which may be,e.g., a still image (photo) or a video image, with images from one ormore databases.

DESCRIPTION OF THE RELATED ART

With the advent of networking technologies, multiple databases have beencreated that contain information and images of individuals, such ashuman resources personnel databases, picture databases, socialnetworking sites (SNS), social media sites, internet marketplaces, andthe like. Such databases and technologies can be used to enable peopleto interact with each other via databases of personal profiles, such asthrough social networking sites (SNS) like Facebook® and Twitter®,social media sites like YouTube® and Tripadvisor®, and marketplaces likeeBay® and Airbnb®. For instance, in certain of the aforementionedexamples a member or user creates a personal profile, which may beposted on a virtual bulletin board. Each personal profile typicallyincludes text and video/photos that are uploaded from the member'scomputer and/or phone. In general, a site has a database that stores theuser profile data of the members and helps the members tofind/communicate with their contacts, who are generally called“friends.”

Most internet based services allow users to easily create a profile.However, the information entered by users is usually unverified. Thisenables malicious users to easily create multiple fake identities fornefarious purposes such as spamming, defrauding or free-riding. It isvaluable for an internet based service to identify real/honest users vs.fake/malicious users. If a service is able to determine the identity ofa person in an image, it allows that service to, e.g., identify fakeidentities that may be using the same fake photo in their image, andidentity real identities by comparing their profile images to imagesfrom other external databases.

Some approaches have thus been attempted to identify the member in agiven query image in a first database by matching the facial image inthe query image to an image stored in a second database, such as asecond SNS. However, the conventional approaches frequently fail. Forexample, existing methods do not work if the member did not upload animage of the member to the second database. Also, even if the memberuploaded the photo, the query image might be taken at a different angleand/or environment than the photos in the second database, significantlyreducing the success rate of the identification process. Moreover, sincea typical image database, such as an SNS database, includes a largenumber of members and each member posts multiple photos, the matchingprocess may require an intractable amount of computational resources ortime. In addition, it is not uncommon that a member (e.g., a user of anSNS) includes something other than a photo of themselves when creatingtheir profile, such as use an image of a celebrity as their profilepicture instead of a picture of themselves. Such cases are particularlyproblematic to current approaches that employ straightforward photomatching. In such a case, the conventional approaches would not workproperly since a large number of members may post photos of thecelebrity on their profiles and matching would indicate that they areall the same user, when in fact, they are different users. There is aneed for systems and methods for determining identity of a person in aquery image with enhanced accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

References will be made to embodiments of the present disclosure,examples of which may be illustrated in the accompanying figures. Thesefigures are intended to be illustrative, not limiting. Although thepresent disclosure is generally described in the context of theseembodiments, it should be understood that it is not intended to limitthe scope of the present disclosure to these particular embodiments.

FIG. 1 shows a schematic diagram of a network environment according toembodiments of the present disclosure.

FIG. 2 shows a query image in a first database and images in a seconddatabase that are compared with the query image according to embodimentsof the present disclosure.

FIG. 3 shows a functional block diagram of a face identifier accordingto embodiments of the present disclosure.

FIG. 4 shows a schematic diagram of a k-dimensional tree (also referredto as a k-d tree) according to embodiments of the present disclosure.

FIG. 5 shows a flowchart of an illustrative process for identifying aperson in a query image according to embodiments of the presentdisclosure.

FIG. 6 shows a flowchart of an illustrative process for reducing searchspace according to embodiments of the present disclosure.

FIG. 7 shows a flowchart of an illustrative process for generating aprincipal component analysis (PCA) feature vector according toembodiments of the present disclosure.

FIG. 8 shows a flowchart of an illustrative process for training adecision tree according to embodiments of the present disclosure.

FIG. 9 shows a computing system according to embodiments of the presentdisclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In the following description, for purposes of explanation, specificdetails are set forth in order to provide an understanding of thepresent disclosure. It will be apparent, however, to one skilled in theart that the present disclosure can be practiced without these details.Furthermore, one skilled in the art will recognize that embodiments ofthe present disclosure, described below, may be implemented in a varietyof ways, such as a process, an apparatus, a system, a device, or amethod on a tangible computer-readable medium.

Components, or nodes, shown in diagrams are illustrative of exemplaryembodiments of the disclosure and are meant to avoid obscuring thedisclosure. It shall also be understood that throughout this discussionthat components may be described as separate functional units, which mayhave sub-units, but those skilled in the art will recognize that variouscomponents, or portions thereof, may be divided into separate componentsor may be integrated together, including integrated within a singlesystem or component. It should be noted that functions or operationsdiscussed herein may be implemented as components. Components may beimplemented in software, hardware, or a combination thereof.

Furthermore, connections between networks and clients within the figuresare not intended to be limited to direct connections. It shall also benoted that the terms “connected” or “communicatively coupled” shall beunderstood to include direct connections, indirect connections throughone or more intermediary devices, and wireless connections.

Furthermore, one skilled in the art shall recognize: (1) that certainsteps may optionally be performed; (2) that steps may not be limited tothe specific order set forth herein; and (3) that certain steps may beperformed in different orders, including being done contemporaneously.

Reference in the specification to “one embodiment,” “an embodiment,” or“embodiments” means that a particular feature, structure,characteristic, or function described in connection with the embodimentis included in at least one embodiment of the disclosure and may be inmore than one embodiment. The appearances of the phrases “in oneembodiment,” “in an embodiment,” or “in embodiments” in various placesin the specification are not necessarily all referring to the sameembodiment or embodiments.

The use of certain terms in various places in the specification is forillustration and should not be construed as limiting. A service ordatabase is not limited to a single service or database; usage of theseterms may refer to a grouping of related services or databases, whichmay be distributed or aggregated.

FIG. 1 shows a schematic diagram of a network environment 100 accordingto embodiments of the present disclosure. As depicted in FIG. 1, one ormore image databases 104 and 106, are connected to a network 102, suchas the Internet. In embodiments, image databases 104 and 106 may be anytype of database that includes image(s) of the users, such as humanresources databases, government databases (e.g., DMV databases), picturedatabases, social networking sites (SNS), and the like. In theillustrated embodiment, image databases 104 and 106 are depicted as SNSdatabases, though should be apparent to those of ordinary skill in theart that other suitable types of databases that include images may beused in place of the SNS. The databases 104 and 106 may allow the usersto create user profiles 114 a-114 i and 116 a-116 j. Even though onlytwo facial image databases (or in short, face databases or databases)are shown in FIG. 1, it should be apparent to those of ordinary skill inthe art that any number of databases may be connected to the network102. It should also be noted that reference to a face database should beunderstood that the database may include more than just images.

The users (or, equivalently members) may use a client devices 108 a-108n (or, in short, clients) to create the user profiles, upload photos orvideo, and otherwise use the services of the site or service (such as,send messages, keep in touch with friends, engage services, etc.). Theterm “user profile” refers to some or all data associated with a user,typically at least part of which is provided as part of establishing anaccount with an SNS or other service or site, that helps identify theuser; such information may include unique identifiers, personalinformation, photos, videos, messages so on, some of which may have beencreated and uploaded to the user's account via one or more clients 108a-108 n. A client 108 a-108 n may be a desktop computer, laptopcomputer, mobile device, or other computing device that has a capabilityto communicate data with the SNSs 104 and 106 via the network 102. Theterms “photo” and “image” may be used interchangeably and refer todigital image from which features are extracted. Also, the term “face”refers to a whole face or a partial face.

In embodiments, each SNS, e.g. 106, may include one or moreservers/computers and software programs that allow the users tocommunicate with other users. The SNS 106 may also include a database(s)for storing data of the user profiles 116 a-116 j. A user may create auser profile in more than one SNS and/or may create multiple userprofiles in the same SNS. In embodiments, the SNS 106 may allocate avirtual bulletin board to each user so that the user can leaveinformation, such as message, photos, and video, on the virtual bulletinboard and the other members of the SNS 106 may access the informationposted on the virtual bulletin board.

In embodiments, one or more photos associated with a user profile, e.g.116 a, may be uploaded from one or more of the clients 108 a-108 n. Inembodiments, the bulletin board may have an interactive album featurethat allows the other members to comment on the photos and identify(tag) people in the photos. In embodiments, the photos may be publishedin a news feed, which is distributed in real-time to the other members.

In embodiments, each SNS may provide a range of privacy options to itsmembers. Each member may make all his communications accessible to everyother member, he may block specific connections or he may keep all hiscommunications private. In embodiments, each member may choose whetheror not to be searchable, decide which parts of his profile are public,decide what not to put in his news feed and determine exactly who cansee its posts.

In embodiments, the SNS 104 may identify a person in a query image usingthe photos that are stored in the SNS 106. FIG. 2 shows a query image204 in the SNS 104 and images 206 a-206 m that are stored in the SNS 106and compared to the query image according to embodiments of the presentdisclosure.

In embodiments, the face identifier 220 may identify the person in thequery image 204 by comparing it against the photos 206 a-206 m in theSNS 106 via the network 102. In embodiments, the face identifier 220 mayextract features of the query image and generate a feature vector thatincludes the extracted features. Hereinafter, this feature vector istermed as an original feature vector. In embodiment, the length of theoriginal feature vector may be 128 bytes, and each byte may correspondto a feature, although other vector sizes may be used. In embodiments,the distance between two feature vectors in the feature vector spacerepresents a level of match between two faces corresponding to the twofeature vectors, i.e., the closer the two feature vectors are in thevector space, the more likely the persons in the two photos are the sameperson.

In embodiments, each of the query image 204 and images 206 a-206 m maybe a still image/photo, a video, or a still frame taken from a video,and the face identifier 220 may identify the person in the query imageby comparing it against the images 206 a-206 m in the SNS 106. In thefollowing sections, for the purpose of illustration, the query image 204and images 206 a-206 are assumed to be photos. However, it should beapparent to those of ordinary skill in the art that the face identifier220 may be used to identify a person in a photo or a still frame takenfrom a video.

The face identifier 220 may compare the original feature vector of thequery image 204 with the original feature vectors for the photos 206a-206 m. If one of the photos 206 a-206 m (e.g., 206 b) matches thequery image, the user profile associated with the photo 206 b may beused to identify the person in the query image 204. In embodiments, theface identifier 220 may find the match by selecting a photo that isclosest to the query image 204 in the original feature vector space,although other classifications of matching may be used. This approachmay be useful when the original feature vector of the photo 206 b isvery close to the original feature vector of the query image 204 in thefeature vector space while the original feature vectors of other photos206 a and 206 c-206 m are far away from the original feature vector ofthe query image 204.

The face identifier 220 may be applied to identify a person in a queryimage by finding a match in a database in which the individual in thephoto and the identity of the associated individual have a high degreeof accuracy/trustworthiness, such as a picture directory, a driver'slicense database, and the like. In such a case, the face in the queryimage may be identified with high precision since it is unlikely thatthere are going to be duplicates of a face in the database, because suchdatabases are highly curated (e.g., it is unlikely that a single personwill have multiple different drivers' licenses, etc.). However, thequery image 204 may have been taken at a different angle and/or have adifferent size from the photo 206 b, which may increase the possibilitythat the face identifier 220 misses the match. In other cases, it may bepossible that the face in the query image 204 may be similar to acelebrity whose photos are uploaded by a large number of members of theSNS 106 in their user profiles. In such cases, the original featurevectors of the photos 206 are clustered, making it difficult to find acorrect match. Also, each of the members of the SNS 106 may uploadmultiple photos, which may significantly increase the number of photos206 to be processed by the face identifier 220. To address suchproblems, in embodiments, the face identifier 220 may process theoriginal feature vectors before inputting to a trained decision treethat calculates a match probability value.

FIG. 3 shows a functional block diagram 300 of a face identifier 301according to embodiments of the present disclosure. As depicted, theface identifier 301 may include: a feature extractor 302 for extractingfeatures from images; a principal component analysis (PCA) map 304 forreducing the dimension of extracted feature vectors; a filter 305 forselecting a set of candidate features vectors from among a large numberof feature vectors (i.e., the filter helps reduce the search space); anda trained decision tree 310 for calculating a probability of match andreturning a match if a match condition is met (e.g., if the probabilityexceeds a threshold value).

In embodiments, the feature identifier 301 may be communicativelycoupled to a database (e.g., an SNS), such as by the network 102, orincluded in the database. For a given query image 204 that includes aface to be identified, the feature extractor 302 extracts features ofthe face and generate an original feature vector for the face in thequery image. In embodiments, OpenFace(https://cmusatyalab.github.io/openface/), which is an open source faceextraction library, may be used to obtain the original feature vector,although other feature extractors may be used. In embodiments, thelength of the original feature vector may be 128 bytes, although othervector sizes may be used.

Because an image database may store huge numbers of images, possiblyeven billions of images, the length of the original feature vectors ofthe images may affect the amount of computational time and computerresources (e.g., data storage) for the identification process.Accordingly, in embodiments, the dimensions of the original featurevectors may be reduced by using principal component analysis (PCA). Inembodiments, the PCA map 304 is used to reduce the dimensionality of theoriginal feature vectors, thereby reducing storage requirements andcomputation resources, but does not significantly compromise robustnessand accuracy in the matching process. In embodiments, the length of theoriginal feature vector may be reduced to 32 bytes, i.e., only 32features among 128 features may be selected and used in identifying theface. Hereinafter, the phrase “PCA feature vector” refers to vectoroutput from a PCA map, such as PCA map 304, which has a dimensionalitythat is less than the dimensionality of the corresponding originalfeature vector. Also, the phrase “query feature vector” may, dependingupon an embodiments, refer to an original feature vector or a PCAfeature vector of a query image, such as the query image 204.

In embodiments, the PCA map 304, which performs dimensionalityreduction, may be trained in advance. The PCA map 304 is trained using aset of original feature vectors as input. Based on the input vectors,the PCA map can determine a dimensionality reduction of the originalfeature vector that does not significantly compromise the robustness indifferentiating the original feature vectors from each other. That is, atradeoff between the number of dimensions of the PCA feature vectors andthe robustness in differentiating the original feature vectors from eachother may be considered when setting a final PCA feature vector size.

FIG. 7 shows a flowchart 700 of an illustrative process for generating aPCA feature vector according to embodiments of the present disclosure.At step 702, a set of profile images may be collected or sampled. Inembodiments, a subset of images (e.g., 80,000 images) may be selectedfrom among a large number of images in the face database 106, and a setof original feature vectors corresponding to the subset of images may begenerated by the feature extractor 302.

At step 704, a PCA map is trained using the set of original featurevectors. Then, in embodiments, a dimension count value, d, is selected(706), where the dimension count d is less than or equal to thedimension of the original feature vectors. In embodiments, a set of testvector pairs are generated (708) by transforming a set of originalfeature vectors into corresponding PCA feature vectors comprising thetop d PCA dimensions, i.e., a set of the original feature vectors aretransformed to their top d PCA dimensions. In embodiments, vectors pairsare formed (e.g., pairs of original feature vector and theircorresponding PCA feature vector). In embodiments, for each test vectorpair, a relative error may be determined (710). For example, inembodiments, the relative error may be defined as:relative error=|true_distance−est_distance|/true_distancewhere true_distance is the distance between the two original featurevectors and est_distance is the distance between corresponding two PCAfeature vectors.

In embodiments, a check is made (712) whether relative error is withinan acceptable range. In embodiments, the acceptable range may be userdefined and may be set such that at each test vector pair's relativeerror must be within an acceptable range, a cumulative value of multipletest vector pairs' relative errors (e.g., a mean, median, or mode) mustbe within an acceptable range, or some combination thereof. One skilledin the art shall recognize that a number of ways may be used forassessing acceptability (e.g., the maximum relative error, the relativeerror at the 95th percentile, etc.), and none of which is critical.

In embodiments, if the error is within an acceptable range, the value ofthe dimension count d may be reduced (714) to examine whether even fewerfeatures may be used to represent an image without significant loss inprecision and recall. Then, in embodiments, steps 708-712 may berepeated until an unacceptable relative error level is observed.

In embodiments, when a dimensionality value, d, has been identified thatproduces an unacceptable level of error, the value of d may be set asthe prior value (i.e., the smallest d value of PCA features thatproduced an acceptable error level at step 712).

In embodiments, all of the original feature vectors may be transformedin to PCA feature vectors using the top d PCA dimensions and stored in aPCA feature database 330.

In embodiments, the PCA feature vectors may be further reduced in size.For example, to save the storage space of the PCA feature database 330,each value of each component (or dimension) of the PCA feature vectormay be discretized and stored as a byte so that each PCA feature vectorin the PCA feature database 330 may be d-bytes long. For instance, ifthe dimension values of a dimension of a PCA feature vector has valuesthat range between −0.8 and 0.8, then, the values may be linearlyinterpolated from [−1.0, 1.0] to the discrete range {0, 1, 2, . . . ,255} so that the dimension is represented as a single byte.

In embodiments, the PCA feature database 330 may be located in the SNS106 or coupled/connected to the SNS 106 via the network 102. When thePCA feature vector for the query image 204 is generated, the filter 305may access the PCA feature database 330 and narrow down the searchspace.

In embodiments, if the query image 204 has one or more index parameters,the prefilter 306 may filter out the PCA feature vectors in the PCAfeature database 330 using an index parameter or parameters to therebynarrow down the search space. For instance, if the name of the person inthe query image 204 is known to be “John Smith” (for example because theimage is associated with a user profile with that name), the prefilter306 may select only the feature vectors associated with “John Smith”among the PCA feature vectors in the database 330. In embodiments, thePCA feature vectors of images, which are associated with user profilesthat are associated with a “John Smith” identifier or index, may beretrieved from the PCA feature database 330 as candidate featurevectors. In embodiments, the candidate PCA feature vectors may beextended to include feature vectors of images from user profilesassociated with John Smith, such as his friends. It is noted that othersuitable index parameters, such as hobby, approximate name, geographiclocation, geo matching, or other data or identifiers, may be used toselect candidate feature vectors from the PCA feature database 330. Itshould be noted that, in embodiments, the phrase “candidate featurevectors” refers to one or more PCA feature vectors but that in otherembodiments it may be other features vectors, such as an originalfeature vector.

In embodiments, other prefiltering may alternatively or additional beperformed. For example, in embodiments, if the query image 204 does nothave any index parameter, one or more k-d trees 308 may be used as aprefilter 306 to search over the PCA feature database 330 and selectcandidate feature vectors. Each of the k-d trees (which is short fork-dimensional tree) 308 is a space-partitioning data structure fororganizing points in a k-dimensional space, where k is an integer lessthan or equal to the dimension of the PCA feature vector.

FIG. 4 shows a schematic diagram of a k-d tree 400 that may be used inembodiments of the present disclosure. The k-d tree 400 is a useful datastructure for searches involving a multidimensional search key and aspecial case of a binary space partitioning tree. As depicted, inembodiments, the k-d tree 400 may include multiple levels, level a-levelj, and each level may include one or more nodes. For instance, the k-dtree 400 may include a root node 402 a at level-a and leaf nodes 402 j1-402 jn at level-j. In embodiments, each node may be associated with acondition for one of the elements in the feature vector.

In embodiments, the k-d tree 400 may be generated before it is used toreduce the search space, i.e., the PCA feature vectors of the images 206a-206 m may be added to the k-d tree 400 in advance. In embodiments,each PCA vector may be added by traversing the k-d tree 400: startingfrom the root node 402 a and moving to either left or right at eachchild node until it reaches one of the vector groups 405 a-405 m. Inembodiments, a library like FLANN (Fast Library for Approximate NearestNeighbors) is used to train a set of k-d trees for answeringnearest-neighbor queries.

It is noted that more than one k-d tree may be used by the filter 305.In embodiments, the number of the k-d trees 308 (and/or the number ofnodes in each k-d tree) may be a function of the size of the PCA featuredatabase 330 and distribution of the PCA feature vectors in the database330. In embodiments, the number of the k-d trees 308 and the number ofsearches in each tree may be balanced so as to avoid going down to awrong branch in the k-d trees 308 and omitting the correct match. Inembodiments, using the k-d trees 308 that are prepared in advance, thefilter 305 may select a vector group (e.g., 405 c) that corresponds tothe query feature vector, where the query vector refers to the PCAfeature vector of the query image 204. In embodiments, the vectors inthe selected vector group 405 c are considered as candidate featurevectors.

In embodiments, the candidate feature vectors selected by the filter 305may be input to a distribution calculator 309. In embodiments, thedistribution calculator 309 may calculate the distance between the queryfeature vector and each of the candidate feature vectors that areselected among the PCA feature database 330 by the filter 305. Also, thedistribution calculator 309 may select a set of shortest distances amongthe distances between the query feature vector and the candidate featurevectors. Then, in embodiments, using the set of shortest distances, thedistribution calculator 309 may generate a decision tree feature vector,where the decision tree feature vector includes the set of shortestdistances and ratios of the set of shortest distances.

For the purposes of illustration, the number of shortest distances isset at four, i.e., the distribution calculator 309 may select the topfour shortest distances, although different numbers may be used. Also,it is assumed, for illustration purposes, that the four shortestdistances are [2, 7, 7, 8], with d₁ being the smallest of the fourdistances. Then, in embodiments, the decision tree feature vector, X,generated by the distribution calculator 309 may be X=[2, 7, 7, 8, 7/2,7/2, 8/2], where the last three components of the decision tree featurevector represent the ratios of the three distances to the first distance(i.e., X=[d₁, d₂, d₃, d₄, d₂/d₁, d₃/d₁, d₄/d₁]. One of the motivationsfor including the ratios in the decision tree feature vector is that,for a correct match, these ratios should be large while, for a falsematch or an inconclusive match, they tend to be small, i.e., the featurevectors are clustered. In embodiments, if the decision tree vectorincludes N feature vectors, the dimension of the distance feature vectormay be (2*N)−1.

In embodiments, the decision tree feature vector generated by thedistribution calculator 309 may be input to the decision tree, which maybe a trained model, 310. In embodiments, the decision tree 310calculates a score using the decision tree feature vector, where thescore represents the probability that a correct match is found (i.e.,the score represents the match probability value). If the score exceedsa threshold value, it is determined that the person in the query image204 is likely the same person as the person associated with featurevector that produced the shortest distance in the decision tree featurevector. In embodiments, if the nearest PCA feature vector in the PCAfeature database is a correct match to the query feature vector, theface identifier 301 may define a label, y, to be positive (“1”).Otherwise, the face identifier 301 may define the label to negative(“0”). In embodiments, the face identifier 301 may generate a pair {X,y}, where the X is the decision tree feature vector and y is the label.

Thus, in embodiments, if the decision tree 310 determines that a featurevector associated with an image, e.g. 206 b, in the SNS 106 is a correctmatch, the identity 312 of the person in the query image 204 may beobtained from the user profile associated with the photo 206 b, and/orprofiles in different network may be linked due to the matching ofimages associated with the corresponding profiles.

In embodiments, the decision tree 310 may be trained in advance. FIG. 8shows a flowchart 800 of an illustrative process for training a decisiontree according to embodiments of the present disclosure. At step 802, aset of profile images associated with persons having known identitiesmay be collected, which will form a ground truth dataset. For example, acorpus of known matched photos between two social networks, Network Aand Network B (e.g., Twitter and Facebook) may be used where there areclear indicators that match the profiles between the two networks, suchas unique identifiers (like email address, Twitter handle, etc.) or theuser has posted a link in one profile (e.g., Network A profile) to theircorresponding profile in the other network (e.g., Network B profile).Since the identities of the faces in the images of the ground truth dataare already known (or at least that they are of the same person), theoperator of the decision tree 310 may train the decision tree byinputting the decision tree feature vectors associated with the imagesof the ground truth data into the decision tree.

At step 804, the decision tree vectors for the set of profile images maybe generated, where each decision tree vector includes the top Nshortest distances, as discussed above in conjunction with thedistribution calculator 309. If the nearest feature vector for an imagein Network A is the correct match for the corresponding image in NetworkB, then a label, y, is assigned a positive (e.g., y_(i)=1); otherwise,the label is assigned a negative value (e.g., y_(i)=0). In embodiments,the face identifier 301 may generate a pair {X_(i), y_(i)}, where the Xis the decision tree feature vector and y is the label. This set offeature vector-label pairs forms a ground truth dataset that may be usedto train (806) the decision tree 310. It should be noted that standardsoftware libraries may be used to train the decision tree to predict ascore for a given distance distribution X. In embodiments, for thetrained decision tree, a score threshold may be chosen by making aprecision-recall plot, or the score may be used directly as theprobability that the matched face is correct.

FIG. 5 shows a flowchart 500 of an illustrative process for identifyinga person in a query image according to embodiments of the presentdisclosure. At step 502, the feature extractor 302 may extract featuresof the face in the query image 204 to generate an original featurevector of the face in the query image. In embodiments, the dimension ofthe original feature vector may be reduced (504) by the PCA map 304. Inembodiments, the vectors generated by the PCA map 304, which arereferred to as PCA feature vectors, have a dimension that is less thanthe original feature vectors inputted to the PCA map.

In embodiments, a filter 305 may be used (506) to narrow the searchspace so that candidate feature vectors can be selected from the largenumber of PCA feature vectors stored in the PCA feature database 330. Inembodiments, using an index parameter associated with the query image204, the prefilter 306 may filter out the PCA feature vectors in the PCAfeature database 330 to thereby select candidate feature vectors fromthe PCA feature database 330. In embodiments, one or more k-d trees 308may be used to select candidate feature vectors from the PCA featuredatabase 330. One skilled in the art shall recognize that otherfiltering techniques may be alternatively and/or additionally employed.

In embodiments, a distribution calculator 309 calculates (508) thedistance between the query feature vector and each of the candidatefeature vectors. In embodiments, the distribution calculator 309 alsoselects a set of shortest distances among the distances between thequery feature vector and the candidate feature vectors. Next, inembodiments, the distribution calculator 309 generates (512) a decisiontree feature vector that includes the set of shortest distances and adistribution of the set of shortest distances.

In embodiments, the decision tree feature vector is input (514) to atrained decision tree 310 to obtain a score for the decision treefeature vector. In embodiments, the decision tree computes a score thatrepresents the probability that a correct match is found. Responsive tothe score exceeding a threshold value, it is determined (516) that theperson in the query image 204 is the same person associated with thefeature vector in the decision tree feature vector that had the smallestdistance. In embodiments, if the nearest PCA feature vector is a correctmatch to the query feature vector, the identity of the person in thequery image 204 may be obtained from the corresponding user profileassociated with that PCA feature vector.

FIG. 6 shows a flowchart 600 of an illustrative process for reducingsearch space at step 506 according to embodiments of the presentdisclosure. At step 602, it is determined whether the query image 204has an index parameter. If the query image 204 has an index parameter,the prefilter 306 may filter out the PCA feature vectors in the PCAfeature database 330 to thereby select candidate feature vectors fromthe PCA feature database 330 at step 604. Then, the process proceeds tostep 508. If the query image 204 does not have any index parameter, oneor more k-d trees 308 may be used to select candidate feature vectorsfrom the PCA feature database 330 at step 606.

It is noted that the person in the query image 204 may be identifiedusing the photos in the SNS 104 instead of the photos 206 a-206 m in theSNS 106. For instance, one user may have created more than one userprofile under slightly different names. In such a case, theidentification process described in conjunction with FIGS. 2-8 may beused to find duplicates/matches of the query image 204 in the SNS 104where the query image 204 is located.

In embodiments, one or more computing system may be configured toperform one or more of the methods, functions, and/or operationspresented herein. Systems that implement at least one or more of themethods, functions, and/or operations described herein may have anapplication or applications operating on at least one computing system.The computing system may have one or more computers and one or moredatabases. The computer system may be a single system, a distributedsystem, a cloud-based computer system, or a combination thereof.

It shall be noted that the present disclosure may be implemented in anyinstruction-execution/computing device or system capable of processingdata, including, without limitation phones, laptop computers, desktopcomputers, and servers. The present disclosure may also be implementedinto other computing devices and systems. Furthermore, aspects of thepresent disclosure may be implemented in a wide variety of waysincluding software (including firmware), hardware, or combinationsthereof. For example, the functions to practice various aspects of thepresent disclosure may be performed by components that are implementedin a wide variety of ways including discrete logic components, one ormore application specific integrated circuits (ASICs), and/orprogram-controlled processors. It shall be noted that the manner inwhich these items are implemented is not critical to the presentdisclosure.

Having described the details of the disclosure, an exemplary system 900,which may be used to implement one or more aspects of the presentdisclosure, will now be described with reference to FIG. 9. Asillustrated in FIG. 9, system 900 includes a central processing unit(CPU) 901 that provides computing resources and controls the computer.CPU 901 may be implemented with a microprocessor or the like; system 900may also include one or more graphics processors and/or floating pointcoprocessors for mathematical computations. System 900 may also includea system memory 902, which may be in the form of random-access memory(RAM) and/or read-only memory (ROM).

A number of controllers and peripheral devices may also be provided, asshown in FIG. 9. An input controller 903 represents an interface tovarious input device(s) 904, such as a keyboard, mouse, or stylus. Theremay also be a scanner controller 905, which communicates with a scanner906. System 900 may also include a storage controller 907 forinterfacing with one or more storage devices 908 each of which includesa storage medium such as magnetic tape or disk, or an optical mediumthat might be used to record programs of instructions for operatingsystems, utilities and applications which may include embodiments ofprograms that implement various aspects of the present disclosure.Storage device(s) 908 may also be used to store processed data or datato be processed in accordance with the present disclosure. System 900may also include a display controller 909 for providing an interface toa display device 911, which may be a cathode ray tube (CRT), a thin filmtransistor (TFT) display, or other type of display. System 900 may alsoinclude a printer controller 912 for communicating with a printer 913. Acommunications controller 914 may interface with one or morecommunication devices 915, which enables system 900 to connect to remotedevices through any of a variety of networks including the Internet, anEthernet cloud, an Fiber Channel over Ethernet (FCoE)/Data CenterBridging (DCB) cloud, a local area network (LAN), a wide area network(WAN), a storage area network (SAN) or through any suitableelectromagnetic carrier signals including infrared signals.

In the illustrated system, all major system components may connect to abus 916, which may represent more than one physical bus. However,various system components may or may not be in physical proximity to oneanother. For example, input data and/or output data may be remotelytransmitted from one physical location to another. In addition, programsthat implement various aspects of this disclosure may be accessed from aremote location (e.g., a server) over a network. Such data and/orprograms may be conveyed through any of a variety of machine-readablemedium including, but are not limited to: magnetic media such as harddisks, floppy disks, and magnetic tape; optical media such as CD-ROMsand holographic devices; magneto-optical media; and hardware devicesthat are specially configured to store or to store and execute programcode, such as application specific integrated circuits (ASICs),programmable logic devices (PLDs), flash memory devices, and ROM and RAMdevices.

Embodiments of the present disclosure may be encoded upon one or morenon-transitory computer-readable media with instructions for one or moreprocessors or processing units to cause steps to be performed. It shallbe noted that the one or more non-transitory computer-readable mediashall include volatile and non-volatile memory. It shall be noted thatalternative implementations are possible, including a hardwareimplementation or a software/hardware implementation.Hardware-implemented functions may be realized using ASIC(s),programmable arrays, digital signal processing circuitry, or the like.Accordingly, the “means” terms in any claims are intended to cover bothsoftware and hardware implementations. Similarly, the term“computer-readable medium or media” as used herein includes softwareand/or hardware having a program of instructions embodied thereon, or acombination thereof. With these implementation alternatives in mind, itis to be understood that the figures and accompanying descriptionprovide the functional information one skilled in the art would requireto write program code (i.e., software) and/or to fabricate circuits(i.e., hardware) to perform the processing required.

It shall be noted that embodiments of the present disclosure may furtherrelate to computer products with a non-transitory, tangiblecomputer-readable medium that have computer code thereon for performingvarious computer-implemented operations. The media and computer code maybe those specially designed and constructed for the purposes of thepresent disclosure, or they may be of the kind known or available tothose having skill in the relevant arts. Examples of tangiblecomputer-readable media include, but are not limited to: magnetic mediasuch as hard disks, floppy disks, and magnetic tape; optical media suchas CD-ROMs and holographic devices; magneto-optical media; and hardwaredevices that are specially configured to store or to store and executeprogram code, such as application specific integrated circuits (ASICs),programmable logic devices (PLDs), flash memory devices, and ROM and RAMdevices. Examples of computer code include machine code, such asproduced by a compiler, and files containing higher level code that areexecuted by a computer using an interpreter. Embodiments of the presentdisclosure may be implemented in whole or in part as machine-executableinstructions that may be in program modules that are executed by aprocessing device. Examples of program modules include libraries,programs, routines, objects, components, and data structures. Indistributed computing environments, program modules may be physicallylocated in settings that are local, remote, or both.

One skilled in the art will recognize no computing system or programminglanguage is critical to the practice of the present disclosure. Oneskilled in the art will also recognize that a number of the elementsdescribed above may be physically and/or functionally separated intosub-modules or combined together.

It will be appreciated to those skilled in the art that the precedingexamples and embodiments are exemplary and not limiting to the scope ofthe present disclosure. It is intended that all permutations,enhancements, equivalents, combinations, and improvements thereto thatare apparent to those skilled in the art upon a reading of thespecification and a study of the drawings are included within the truespirit and scope of the present disclosure. It shall also be noted thatelements of the claims, below, may be arranged differently includinghaving multiple dependencies, configurations, and combinations.

What is claimed is:
 1. A method for detecting whether a personassociated with a query image is a same person associated with anotherimage in a set of images, the method comprising: generating a queryfeature vector for a query image, the query feature vector includingfeatures extracted from the query image; determining a distance betweenthe query feature vector and each of a set of candidate feature vectorsthat correspond to the set of images; selecting a set of shortestdistances among the determined distances; generating a feature vectorcomprising the set of shortest distances and a distribution of the setof shortest distances; inputting the feature vector to a trained modelto obtain a match probability value; and responsive to the matchprobability value exceeding a threshold value, determining the personassociated with the query image is the same person associated with animage corresponding to the candidate feature vector that has theshortest distance among the set of shortest distances.
 2. The method ofclaim 1, wherein the step of generating a query feature vectorcomprises: generating an original feature vector that includes one ormore features extracted from the query image; and converting theoriginal feature vector into the query feature vector by reducing adimension of the original feature vector.
 3. The method of claim 2,wherein the step of converting the original feature vector comprisesinputting the original feature vector to a principal component analysis(PCA) map to obtain the query feature vector.
 4. The method of claim 1,further comprising, prior to the step of determining a distance:reducing a search space comprising a plurality of feature vectors to getthe set of candidate feature vectors.
 5. The method of claim 4, whereinthe step of reducing a search space comprises: using an index parameterassociated with the query image, selecting the set of candidate featurevectors among the plurality of feature vectors.
 6. The method of claim1, wherein the set of images are included in a social networking service(SNS).
 7. The method of claim 1, further comprising: training the modelusing a set of feature vectors that correspond to a set of imagesassociated with persons having known identities.
 8. A system fordetecting whether a person associated with a query image is a sameperson associated with another image in a set of images, the systemcomprising: one or more processors; a database for storing the set ofimages and coupled to the one or more processors; a face identifiercommunicatively coupled to the one or more processors and configured to:generate a query feature vector for a query image, the query featurevector including features extracted from the query image; determine adistance between the query feature vector and each of a set of candidatefeature vectors that correspond to the set of images; select a set ofshortest distances among the determined distances; generate a featurevector comprising the set of shortest distances and a distribution ofthe set of shortest distances; input the feature vector to a trainedmodel to obtain a match probability value; and responsive to the matchprobability value exceeding a threshold value, determine the personassociated with the query image is the same person associated with animage corresponding to the candidate feature vector that has theshortest distance among the set of shortest distances.
 9. The system ofclaim 8, wherein the face identifier is further configured to generatean original feature vector that includes one or more features extractedfrom the query image and convert the original feature vector into thequery feature vector by reducing a dimension of the original featurevector.
 10. The system of claim 9, wherein the face identifier uses aprincipal component analysis (PCA) map to convert the original featurevector into the query feature vector.
 11. The system of claim 8, whereinthe face identifier comprises a filter that reduces a search spacecomprising a plurality of feature vectors to get the set of candidatefeature vectors.
 12. The system of claim 11, wherein the filtercomprises a prefilter that uses an index parameter associated with thequery image to select the set of candidate feature vectors among theplurality of feature vectors.
 13. The system of claim 11, wherein thefilter uses one or more k-d trees to select the set of candidate featurevectors among the plurality of feature vectors.
 14. The system of claim8, wherein a distribution of the set of shortest distances comprises oneor more ratios between the set of shortest distances.
 15. Anon-transitory computer-readable medium or media comprising a set ofinstructions for detecting whether a person associated with a queryimage in a first user profile on a first face database is a same personassociated with a second image in a second user profile on a second facedatabase, wherein execution of the set of instructions by one or moreprocessors causes the one or more processors to perform the steps of:generating a query feature vector for a query image in a first userprofile on a first image database, the query feature vector includingfeatures extracted from the query image; determining a distance betweenthe query feature vector and each of a set of candidate feature vectorsthat correspond to a set of images on the second image database;selecting a set of shortest distances among the determined distances;generating a feature vector comprising the set of shortest distances anda distribution of the set of shortest distances; inputting the featurevector to a trained model to obtain a match probability value; andresponsive to the match probability value exceeding a threshold value,determining the person associated with the query image is the sameperson associated with an image corresponding to the candidate featurevector that has the shortest distance among the set of shortestdistances.
 16. The non-transitory computer-readable medium or media ofclaim 15, wherein the step of generating a query feature vectorcomprises: generating an original feature vector that includes one ormore features extracted from the query image; and converting theoriginal feature vector into the query feature vector by reducing adimension of the original feature vector.
 17. The non-transitorycomputer-readable medium or media of claim 16, wherein the step ofconverting the original feature vector comprises inputting the originalfeature vector to a principal component analysis (PCA) map to obtain thequery feature vector.
 18. The non-transitory computer-readable medium ormedia of claim 15, wherein the execution of the set of instructions byone or more processors causes the one or more processors to perform theadditional step of: reducing a search space comprising a plurality offeature vectors to get the set of candidate feature vectors.
 19. Thenon-transitory computer-readable medium or media of claim 18, whereinthe step of reducing a search space comprises: using an index parameterassociated with the query image, selecting the set of candidate featurevectors among the plurality of feature vectors.
 20. The non-transitorycomputer-readable medium or media of claim 18, wherein the step ofreducing a search space comprises: using one or more k-d trees,selecting the set of candidate images among the plurality of featurevectors.